A Minimal Encoding Approach to Feature Discovery

This paper discusses unsupervised learning of orthogonal concepts on relational data. Relational predicates, while formally equivalent to the features of the concept-learning literature, are not a good basis for de ning concepts. Hence the current task demands a much larger search space than traditional concept learning algorithms, the sort of space explored by connectionist algorithms. However the intended application, using the discovered concepts in the Cyc knowledge base, requires that the concepts be interpretable by a human, an ability not yet realized with connectionist algorithms. Interpretability is aided by including a characterization of simplicity in the evaluation function. For Hinton's Family Relations data, we do nd cleaner, more intuitive features. Yet when the solutions are not known in advance, the di culty of interpreting even features meeting the simplicity criteria calls into question the usefulness of any reformulation algorithm that creates radically new primitives in a knowledge-based setting. At the very least, much more sophisticated explanation tools are needed. This paper discusses conceptual clustering using the Minimum Description Length principle in a domain of family relationships rst solved by Hinton using back-propagation. This problem is unsuitable for algorithms such as ID3 because the features used in the problem description are very far removed from those present in an intuitive theory of family relationships. The algorithm described here, like back-propagation, uses constructive induction to discover a new set of features. The goal is then to use them in rules in the Cyc knowledge base to capture some of the domain regularities. Hence the features must be interpretable by a human. This requires maintaining the power of connectionist systems to discover completely new features while enhancing interpretability, which is accomplished by including a characterization of feature simplicity in the feature evaluation function. Indeed no other algorithm has discovered such clean intuitive features for problems, like Family Relations, in which the original data is not naturally represented as feature-vectors. Yet this success has pointed out the di culty of interpreting even very good features, and calls into question the usefulness of any representation discovery algorithm that creates radically new primitives in a knowledge-based setting. At the very least, much more sophisticated explanation tools are needed. This is an abridged version of ACT-CYC-234-90, \The Minimum Description Length Principle Applied to Feature Learning and Analogical Mapping." MCC Non-Con dential ii